Enhanced spatialization system with satellite device

ABSTRACT

A system enhances spatialization in which spatial information about sound sources at an originating location is represented in an audio signal. The system applies a phase difference analysis to the signals received from an array of spaced apart input devices or microphones to derive spatial or directional information about the relative directions of one or more satellite input devices or microphones. The signals from the satellite input devices or microphones are mixed by a function of their respective directions to generate a multichannel output signal. When processed by a remote or local system, the output signal provides a representation of the relative directions of the sound sources at the originating location at a receiving location.

PRIORITY CLAIM

This application claims the benefit of priority from U.S. ProvisionalApplication Nos. 61/301,745 and 61/301,761, both of which were filed onFeb. 5, 2010, which are incorporated by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

This disclosure relates to the field of multichannel communications. Inparticular, to a system that enhances spatialization with a satellitedevice.

2. Related Art

Some voice communication has been carried out over a single audiochannel and often over a narrow band of the audio spectrum, between 200Hz and 3500 Hz. This has made some voice communications difficult tounderstand. When voice communication is paired with a video feed (i.e.,in a video conferencing system) the low quality voice communication cansignificantly degrade the overall user experience.

Some video conferencing systems use an array of microphones. Thephysical arrangement of one microphone, in relation to anothermicrophone, is not fixed and a microphone may be moved while aconference is in progress, for example, to situate them closer toparticular speakers at different times.

The microphone signals from the main microphones and an ancillarymicrophone may be mixed with an emphasis on making all speakersintelligible irrespective of which microphone they are nearest to. Thespatial information is not well represented in the output signals orpresented at the other end of the video conference. This can beconfusing or annoying for participants as the voices coming out ofloudspeakers do not have a spatial layout that corresponds to theapparent positions of the speakers on the video display device.

SUMMARY

A system enhances spatialization in which spatial information aboutsound sources at an originating location is represented in an audiosignal. The system applies a phase difference analysis to the signalsreceived from an array of spaced apart input devices or microphones toderive spatial or directional information about the relative directionsof one or more satellite input devices or microphones. The signals fromthe satellite input devices or microphones are mixed as a function oftheir respective directions to generate a multichannel output signal.When processed, the output signal provides a representation of therelative directions of the sound sources at the originating location.

Other systems, methods, features and advantages will be, or will become,apparent to one with skill in the art upon examination of the followingfigures and detailed description. It is intended that all suchadditional systems, methods, features and advantages be included withthis description, be within the scope of the invention, and be protectedby the following claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The system may be better understood with reference to the followingdrawings and description. The components in the figures are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the invention. Moreover, in the figures, likereferenced numerals designate corresponding parts throughout thedifferent views.

FIG. 1 is a system that enhances spatialization with a satellite inputdevice or satellite microphone.

FIG. 2 is a spatial analysis process.

FIG. 3 is a block diagram of the spatialization system/spatializationprocess within a vehicle.

FIG. 4 is a block diagram of the spatialization system/spatializationprocess within a wireless device/wireless architecture.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A system enhances spatialization among a plurality of directional oromnidirectional inputs devices that may operate in tandem to convertsound into analog signals or digital data. Hardware and software convertand transmit voiced and unvoiced input signals across a wireless (e.g.,radio, microwave, etc.) or physical medium to a system that enables aSignal-To-Noise (SNR) ratio and spatial scalability. The directionalityof the voiced and unvoiced input signals may be identified, mixed, androuted into one or more channels that may be transmitted through awireless or fixed medium to a multi-dimensional output system. Atransmitter may electrically encode data or the signals that are thenradiated or conveyed to an output system. The output system may decodethe received signals at a receiver to derive multiple dimensionalseparations between the directional or omnidirectional inputs. Theoutput may be reproduced to render a substantially originaldirectionality of the received signals or, alternatively, may be scaledto exploit the separation between those signals.

FIG. 1 is a schematic of a system 100 that enhances spatialization. Thesystem 100 may be positioned at an originating location 150 in acommunication arrangement such as, for example, a teleconferencing orvideo conferencing room, an interior of a vehicle (e.g., FIG. 3), orwithin any communication or telecommunication device that processesaudible sound from one or more sound sources, or speakers 152. Thesystem 100 may comprise devices that convert audible sound from one, twoor more sound sources 152 into electrical or optical signals. In somesystems a microphone array 102 and a satellite input device or satellitemicrophone 108 may convert sound into analog, discrete, optical orelectrical signals. The microphone array 102 may, for example, comprisea pair of closely spaced omnidirectional microphones. In some systems100, the omnidirectional microphones may be spaced less than about awavelength apart for frequencies within a spectrum of interest. Thesatellite input device may have an omnidirectional receiving pattern orselectable (e.g., user-selectable) receiving patterns that may favorsound sourced from one or more directions while being less sensitive tosounds traveling from other directions. Additional hardware and softwaresuch as a wireless or tangible serial or parallel interface and bus mayconvey the output signal to a spatial analyzer 104 and an audio mixer106. The spatial analyzer 104 may receive, and in some systems, decodethrough a decoder or converter the signals from each of the microphonesthat may comprise the microphone array 102 and the satellite microphone108. The microphone signals (e.g., that may be processed as channels)are analyzed to derive a directionality such as the directions of soundreceived from the satellite microphone 108 relative to the microphonearray 102. The signals may then be combined, routed, and furtherprocessed to change the level, timbre, and/or dynamics of the audiosignal. An electronic device, such as a transmitter 110 may encode thesignals that may then be conveyed or radiated through a transmissionmedia to another location. In some systems the transmitter 110 conveyssignals through frequency modulation, amplitude modulation, or anywireless communication protocol such as cellular (e.g., Code DivisionMultiple Access (CDMA), Global System for Mobile Communication (GSM),etc.), Bluetooth, WLAN, WiFi, Hyper/LAN, Ultraband, WiMax, Mobile-Fi, orZigBee mobility.

The directionality of the input signals may be derived through thespatial analyzer 104 that may execute or may be configured to executethe functions of a spatial analysis process. An exemplary spatialanalysis process 200 may process the received signal in one, two, ormore communication domains or compressions (e.g., signal strength may berepresented as a function of frequency, time, wavelets, etc.) In FIG. 2,the spatial analysis process 200 may transform the entire or selectedportions of the outputs of the microphone array 102 and satellitemicrophone 108 signals into the frequency domain through 210 and 220.Through spatial analysis hardware or the execution of software, theanalysis process 200 may compute a phase for each or selected microphonearray 102 signals and the satellite microphone 108 signal over a rangeof frequencies. The spatial analysis 200 may calculate a phasedifference between the microphone array 102 signals and satellitemicrophone 108 signals at 230, compute or measure a signal to noiseratio (SNR) for each of one or more frequency bins of interest at 240,and compute or derive a SNR weighted slope of the phase difference as afunction of frequency at 250. Based on the change in phase with respectto the change in frequency, the spatial analysis 200 may derive adirection of the sound(s) received at the satellite microphone 108relative to the sound(s) received at the microphone array 102 throughthe SNR weighted slope. In some two-dimensional processes a positiveslope may indicate that the satellite microphone 108 is positioned at orcloser to one side of the microphone array 102 and a negative slope mayindicate that the satellite microphone 108 is positioned at or closer toa second or an opposite side of the microphone array 102.

To limit sudden changes in derived sound directionality, a smoothingprocess may smooth the identified directionality over time at 260,through a smoothing circuit. The spatial analysis process may calculatethe root-mean-square (RMS) of the amplitude of the satellite microphone108 signal and the microphone array 102 signals, respectively. Thedirectionality of the satellite microphone 108 may be stored in local ordistributed memory devices and updated synchronously or whenpredetermined or user-set conditions are met. In some systems, thedirectionality of the satellite microphone 108 signals are stored whenthe slope of the smoothed direction is above a predetermined or user-setthreshold, the RMS of the amplitude of the satellite microphone 108signal is above a user-set or predetermined threshold, and/or when theRMS of the amplitude of the satellite microphone 108 signal is abouttwice as large as the RMS of the amplitude of the microphone array 102signals. At 270, the spatial analysis process may convert the smoothedsignal to another domain such as a temporal domain at 270, before theanalog or digital signals that comprise the satellite microphone 108signals are routed to one or more audio channels that include theprocessed analog or digital signals that comprise the output of themicrophone array 102. At 280, the satellite microphone 108 signals maybe further processed with the microphone array 102 signals to change thelevel of timber and/or dynamics of the audio signal at 280. In FIG. 1,an audio mixer 106 may mix or proportionally mix the satellitemicrophone 108 signal into one or more channels of a multichannel outputsignal 120 as a function of (or based on) the direction of the input tothe satellite microphone 108 relative to the input to the microphonearray 102 that may be derived through the analysis process 200. Theaudio mixer 106 preserves the content received at the satellitemicrophone 108 with the content received at microphone array 102.

The system 100 may process the voices (or acoustic signal) of far-field,relative to the microphone array 102 and the satellite microphone 108,speakers 152 (or sound sources). The direction of the satellitemicrophone 108 may be smoothed over time to provide a stable identifieddirectionality to the audio mixer 106. The direction may beasynchronously updated in time, for example, when it appears that thesatellite microphone 108 may have been moved relative to the microphonearray 102, during pauses in detected speech, or alternatively, may besynchronously updated in time. A ratio (e.g., SNR) of the energy of thesatellite microphone 108 signal compared to the microphone array 102signals may be processed by a processor or comparator as an indicator ofa satellite microphone 108 movement.

The output signal 120 may be transmitted through a wireless or tangiblemedium to a local or remote receiving location 160 where an audioplayback system 170 converts the transmitted signal 120 (also known as“output signal 120”) into perceptible forms. In FIG. 1, the audioplayback system 170 may convert the output signal 120 into aural signalsnear one or more receivers or listeners 162 through two or more outputdevices such as loudspeakers 175. The listeners 162 may perceive spatialand/or positional information (e.g., the derived direction and/orposition of the satellite microphone 108 relative to the microphonearray 102) in the output of the loudspeakers 175. The voices of eachspeaker 152 proximate to the satellite microphone 108 may be perceivedto come from a position or direction (e.g., through two, three, or moredimensions or coordinate directions such as left, a right, etc.,direction) in the receiving location 160 that is related to theirrelative positions in the originating location 150. The listeners 162may experience a higher quality fidelity (e.g., an enhancedspatialization) in which they are able to associate a relative spatiallocation or position with each of the speakers' 152 voices, which mayfurther improve intelligibility.

The system 100 may be used at a common location (e.g., the originatinglocation 150) of a communication arrangement, at two or more locallocations, or alternatively, may be distributed across some or allremote participating locations in a communication arrangement orcommunication network. To render the spatial information contained in anaural or multimedia output signal 120 from another location (e.g., anoriginating location 150) each terminating location (e.g., eachreceiving location 160) may tangibly or wirelessly interface one or moremultichannel playback systems 170 that interface or couple a pluralityof output devices such as loudspeakers 175.

The system 100 may process two channels (e.g., stereo) through an inputinterface of an audio mixer 106. The audio mixer 106 may receive anddecode the channels through a decoder. The audio playback system 170 mayprocess the stereo output signals and transmit the content through twoor more loudspeakers 175 to render a more natural sound distribution.

The system 100 may be used with one, two, or more satellite microphones108. A direction for each satellite microphone 108 may be derived by aspatial analysis process. The output of each satellite microphone 108may be processed and mixed with the output of microphone array 102. Thesatellite output may be routed to one or more audio channels, and may beprocessed with the array output to change the timbre and/or dynamics ofthe signals. In some systems, a post spatial analysis occurs through amixer that may mix analog or digital signals, depending on the type ofmixer, and then sums the modified signals to produce a combined output.A transmitter 110 may then transmit the output to one or more receivinglocations 160. Besides the hardware implementations that are referencedor in the alternative, the spatial analysis method 200 or functionalitymay be implemented in software retained in a fixed computer readablemedium that may be executed by a processor.

The system, methods, and descriptions described may be programmed in oneor more controllers, devices, processors such as signal processors(e.g., processors 116 and 118) that may execute all or some of thefunctionality (e.g., the system may execute any combination of acts)shown in FIG. 2. The processors may comprise one or more centralprocessing units or digital signal processors that supervise thesequence of micro-operations that execute the instruction code and datacoming from memory (e.g., computer readable medium) that generate,support, and/or complete an operation, compression, or signalmodifications. The dedicated applications may support and define thefunctions of the special purpose processor or general purpose processorthat is customized by instruction code (and in some applications may beresident to vehicles, communication systems, audio systems, telephones,teleconferencing systems, cellular systems, etc.). In alternativeimplementations, the system 100, satellite microphones 108, or some ofall of the acts of the spatial analysis process 200 may be executed orcomprise an integrated or unitary part of a fixed or mobile wirelessdevice, interface an in-vehicle bus, interface a universal serial bus(or buses having a bandwidth of about 1.5 megabits per second orgreater) or interface cables such as an interface to audio or multimediacables. Some or all of the components of the system 100 and/or some orall of the acts of the spatial analysis process 200 may be integratedwithin or comprise a unitary part of a vehicle (e.g., FIG. 3) wirelessdevice (e.g., FIG. 4) such as a smart phone, portable computer, personalmobile computer, a touch screen based device (e.g., Tablet Computers),or comprise part of a wireless architecture, e.g., FIG. 4). In somesystems, a front-end processor may perform the complementary tasks ofgathering data for a processor or program to work with, and for makingthe data and results available to other processors, controllers, ordevices. In some systems, processors 116 and/or 118 may comprise asingle processor that interfaces with, is an integrated part of, or is aunitary part of the spatial analyzer 104.

The systems, methods, and descriptions may program one or more signalprocessors or may be encoded in a signal bearing storage medium, acomputer-readable medium, or may comprise logic stored in a memory thatmay be accessible through an interface and is executable by one or moreprocessors. Some signal-bearing storage medium or computer-readablemedium comprise a memory that is unitary or separate (e.g., local orremote) from a device, programmed within a device, such as one or moreintegrated circuits, or retained in memory and/or processed by acontroller or a computer. If the descriptions or methods are performedby software, the software or logic may reside in an electronic oroptical memory resident to or interfaced to one or more processors,devices, or controllers that may support a tangible or visualcommunication interface (e.g., to a display), wireless communicationinterface, or a wireless system.

The memory may retain an ordered listing of executable instructions in aprocessor, device, or controller accessible medium for implementinglogical functions. A logical function may be implemented through digitalcircuitry, through source code, or through analog circuitry. Thesoftware may be embodied in any computer-readable medium, signal-bearingmedium, or other non-transitory medium for use by, or in connectionwith, an instruction executable system, apparatus, and device, residentto system that may maintain persistent or non-persistent connections.Such a system may include a computer system, a processor-based system,or another system that includes an input and output interface that maycommunicate with a publicly accessible or privately accessibledistributed network through a wireless or tangible communication busthrough a public and/or proprietary protocol.

A “computer-readable storage medium,” “machine-readable medium,”“propagated-signal” medium, and/or “signal-bearing medium” may comprisea medium (e.g., a non-transitory medium) that stores, communicates,propagates, or transports software or data for use by or in connectionwith an instruction executable system, apparatus, or device. Themachine-readable medium may selectively be, but not limited to, anelectronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, device, or propagation medium. Anon-exhaustive list of examples of a machine-readable medium wouldinclude: an electrical connection having one or more wires, a portablemagnetic or optical disk, a volatile memory, such as a Random AccessMemory (RAM), a Read-Only Memory (ROM), an Erasable ProgrammableRead-Only Memory (EPROM or Flash memory), or an optical fiber. Amachine-readable medium may also include a tangible medium, as thesoftware may be electronically stored as an image or in another format(e.g., through an optical scan), then compiled, and/or interpreted orotherwise processed. The processed medium may then be stored in acomputer and/or machine memory.

While various embodiments of the invention have been described, it willbe apparent to those of ordinary skill in the art that many moreembodiments and implementations are possible within the scope of thepresent invention. Accordingly, the invention is not to be restrictedexcept in light of the attached claims and their equivalents.

1. A system that enhances spatialization with a satellite microphonecomprising: an array device that processes first incoming audio signalsand converts the audio signals into electrical signals; a satelliteinput device that processes second incoming audio signals received at alocation distant from the array device; and a processor that executes acomputer readable medium comprising: computer program code that analyzesthe first incoming audio signals and the second incoming audio signalsto derive a direction of the second incoming audio signals received bythe satellite input device relative to the first incoming audio signalsreceived by the array of device; computer program code that smoothes thederived direction over time; computer program code that updates thederived direction in a memory when criteria indicate that the satelliteinput device may have been moved; and computer program code thatproportionally mixes an output of a satellite input device signal intoone or more audio channels of a multichannel output signal.
 2. Thesystem of claim 1 where the array device comprise a plurality ofmicrophones.
 3. The system of claim 1 where the electrical signalscomprise discrete signals.
 4. The system of claim 1 where the electricalsignals comprise continuous signals.
 5. The system of claim 1 where thearray device comprises a plurality of omnidirectional microphonesoperating in tandem.
 6. The system of claim 1 further comprisingcomputer program code that transforms the electrical signals from afirst electrical domain to a second electrical domain before theprocessor executes the computer program code that derives the direction.7. A system that enhances spatialization with a satellite microphonecomprising: a plurality of microphones arranged in an array that convertsound waves into electrical signals; a satellite device that processessound waves at a location remote from the plurality of microphones; anda spatial analyzer configured to derive a first directionality of aninput to the satellite device relative to a second directionality of aninput to the plurality of microphones through a comparison of slopesderived from an output of the plurality of microphones and an output thesatellite device.
 8. The system of claim 7 where the spatial analyzer isfurther configured to transform the entire or selected output of theplurality of microphones and the satellite device from a firstelectrical domain to a second electrical domain.
 9. The system of claim7 where the spatial analyzer is further configured to derive a pluralityof signal-to-noise ratios for one or more frequency bins that representthe outputs of the plurality of microphones and the output of thesatellite device.
 10. The system of claim 7 where the slopes comprise aplurality of signal-to-noise weighted slopes.
 11. The system of claim 7where the plurality of microphones comprise a plurality of closelyspaced omnidirectional microphones.
 12. The system of claim 7 where thespatial analyzer is further configured to store the first directionalityof the input of the satellite device relative to the seconddirectionality of the input of the plurality of microphones in a localmemory.
 13. The system of claim 12 where the spatial analyzer is furtherconfigured to store the first directionality of the input of thesatellite device relative to the second directionality of the input tothe plurality of microphones in the local memory upon an occurrence ofan asynchronous event.
 14. The system of claim 7 where the spatialanalyzer is further configured to smooth a plurality of sounddirectionalities derived through the comparison of slopes.
 15. Thesystem of claim 7 further comprising means for mixing a processed outputof the satellite device with a processed output of the plurality ofmicrophones that substantially preserves the first directionality andthe second directionality.
 16. The system of claim 7 further comprisinga mixer configured to process the output of the spatial analyzer thatsubstantially preserves a first content and the first directionality ofthe satellite device and a second content and the second directionalityof the output of the plurality of microphones.
 17. The system of claim16 where the mixer proportionally mixes its input based on the firstdirectionality relative to the second directionality and adjustsdynamics of the output of the spatial analyzer.
 18. The system of claim17 further comprising a transmitter coupled to the mixer that sendselectrically encoded data to a remote location.
 19. The system of claim18 where the spatial analyzer is further configured to derive a positionof the satellite device relative to a position of the plurality ofmicrophones in two-dimensions.
 20. The system of claim 7 where spatialanalyzer is further configured to derive a position of the satellitedevice relative to a position of the plurality of microphonesasynchronously in time.
 21. The system of claim 7 where spatial analyzeris further configured to derive a position of the satellite devicerelative to a position of the plurality of microphones synchronously intime.
 22. A method enhances spatialization with a satellite devicecomprising: converting sound waves into electrical signals receivedthrough a plurality of microphones; processing sound waves through asatellite device at a location remote from the plurality of microphones;and deriving a directionality of an input to the satellite devicerelative to directionality of an input to the plurality of microphonesthrough a comparison of slopes derived from an output of the pluralityof microphones and an output the satellite device, respectively.
 23. Themethod of claim 22 further comprising analyzing a plurality ofsignal-to-noise ratios, a plurality of phases, and a plurality of slopesto derive a position of the satellite device relative to a position ofthe plurality of microphones.
 24. The method of claim 23 furthercomprising smoothing the derived position of the satellite devicerelative to the position of the plurality of microphones andtransmitting data representative of the position of the satellite deviceand the directionality of the input to the satellite device relative tothe plurality of microphones.